Automatic Corpus Selection and Halting Condition Detection for Semantic Asset Expansion

ABSTRACT

A mechanism is provided in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement an automated lexicon expansion for an identified corpus. For a selected corpus in a set of corpora, the mechanism determines an estimated number of new terms in the selected corpus that are not in the lexicon based on a frequency count known terms in the selected corpus. Responsive to the estimated number of new terms in the selected corpus being greater than a threshold, the mechanism performs lexicon expansion using the selected corpus to form an expanded lexicon. Responsive to the estimated number of new terms in the selected corpus not being greater than the threshold, the mechanism halts lexicon expansion.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for automaticcorpus selection and halting condition detection for semantic assetexpansion.

With the increased usage of computing networks, such as the Internet,humans are currently inundated and overwhelmed with the amount ofinformation available to them from various structured and unstructuredsources. However, information gaps abound as users try to piece togetherwhat they can find that they believe to be relevant during searches forinformation on various subjects. To assist with such searches, recentresearch has been directed to generating Question and Answer (QA)systems which may take an input question, analyze it, and return resultsindicative of the most probable answer to the input question. QA systemsprovide automated mechanisms for searching through large sets of sourcesof content, e.g., electronic documents, and analyze them with regard toan input question to determine an answer to the question and aconfidence measure as to how accurate an answer is for answering theinput question.

Examples of QA systems are the IBM Watson™ system available fromInternational Business Machines (IBM®)Corporation of Armonk, N.Y., Siri®from Apple®, and Cortana® from Microsoft®. The IBM Watson™ system is anapplication of advanced natural language processing, informationretrieval, knowledge representation and reasoning, and machine learningtechnologies to the field of open domain question answering. The IBMWatson™ system is built on IBM's DeepQA™ technology used for hypothesisgeneration, massive evidence gathering, analysis, and scoring. DeepQA™takes an input question, analyzes it, decomposes the question intoconstituent parts, generates one or more hypotheses based on thedecomposed question and results of a primary search of answer sources,performs hypothesis and evidence scoring based on a retrieval ofevidence from evidence sources, performs synthesis of the one or morehypotheses, and based on trained models, performs a final merging andranking to output an answer to the input question along with aconfidence measure.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described herein in the DetailedDescription. This Summary is not intended to identify key factors oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

In one illustrative embodiment, a method is provided in a dataprocessing system comprising at least one processor and at least onememory, the at least one memory comprising instructions executed by theat least one processor to cause the at least one processor to implementan automated lexicon expansion for an identified corpus. The methodcomprises, for a selected corpus in a set of corpora, determining anestimated number of new terms in the selected corpus that are not in thelexicon based on a frequency count of known terms in the selectedcorpus. The method further comprises, responsive to the estimated numberof new terms in the selected corpus being greater than a threshold,performing lexicon expansion using the selected corpus to form anexpanded lexicon. The method further comprises, responsive to theestimated number of new terms in the selected corpus not being greaterthan the threshold, halting lexicon expansion.

In other illustrative embodiments, a computer program product comprisinga computer usable or readable medium having a computer readable programis provided. The computer readable program, when executed on a computingdevice, causes the computing device to perform various ones of, andcombinations of, the operations outlined above with regard to the methodillustrative embodiment.

In yet another illustrative embodiment, a system/apparatus is provided.The system/apparatus may comprise one or more processors and a memorycoupled to the one or more processors. The memory may compriseinstructions which, when executed by the one or more processors, causethe one or more processors to perform various ones of, and combinationsof, the operations outlined above with regard to the method illustrativeembodiment.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 depicts a schematic diagram of one illustrative embodiment of aquestion/answer creation (QA) system in a computer network;

FIG. 2 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments are implemented;

FIG. 3 illustrates a QA system pipeline for processing an input questionin accordance with one illustrative embodiment;

FIG. 4 is a block diagram illustrating a mechanism for semantic assetexpansion in accordance with an illustrative embodiment;

FIG. 5 shows an example graph of frequency counts for known terms in agiven corpus in accordance with an illustrative embodiment;

FIG. 6 shows an example graph of frequency counts for known terms in agiven corpus with a line fitted to find the X-axis intercept inaccordance with an illustrative embodiment;

FIG. 7 is a flowchart illustrating operation of a system for automaticcorpus selection and halting condition detection for semantic assetexpansion in accordance with an illustrative embodiment;

FIG. 8 is a flowchart illustrating operation of a mechanism for sortingcorpora by estimated number of new terms in accordance with anillustrative embodiment; and

FIG. 9 is a flowchart illustrating operation of a mechanism fordetermining an estimated number of missing terms in a corpus inaccordance with an illustrative embodiment.

DETAILED DESCRIPTION

Semantic assets, such as lexicons, dictionaries, sets of terms, lists ofitems, are vital ingredients in natural language processing. Theirapplicability range is wide and deep from spotting entities todetermining sentiment. Creation and curation of these semantic assetscan be time consuming if done completely manually. Approaches such asWord2Vec and GLIMPSE enable more rapid creation of such assets byallowing the creator to pick items from a list likely related to theitems that are already in the lexicon rather than creating such listswhole cloth.

Word2Vec is a group of related models that are used to produce wordembeddings. These models are shallow, two-layer neural networks that aretrained to reconstruct linguistic contexts of words. GLIMPSE stands forGLobal IMPlieit Search. While many text indexing schemes create quitelarge indexes (usually around 50% of the size of the original text), aGLIMPSE-created index is only 2-4% of the size of the original text.

When using the Word2Vec and GLIMPSE iterative approach of picking wordsfrom generated related terms, one must determine when to stop adding tothe lexicon. The illustrative embodiments provide a mechanism toquantify the tradeoffs of a stopping point and how many new terms can beexpected in a given dataset. Given this function, one obviousapplication of the mechanism is selection from among several corpora forthe best corpus to use for semantic asset expansion.

Before beginning the discussion of the various aspects of theillustrative embodiments in more detail, it should first be appreciatedthat throughout this description the term “mechanism” will be used torefer to elements of the present invention that perform variousoperations, functions, and the like. A “mechanism,” as the term is usedherein, may be an implementation of the functions or aspects of theillustrative embodiments in the form of an apparatus, a procedure, or acomputer program product. In the case of a procedure, the procedure isimplemented by one or more devices, apparatus, computers, dataprocessing systems, or the like. In the case of a computer programproduct, the logic represented by computer code or instructions embodiedin or on the computer program product is executed by one or morehardware devices in order to implement the functionality or perform theoperations associated with the specific “mechanism.” Thus, themechanisms described herein may be implemented as specialized hardware,software executing on general purpose hardware, software instructionsstored on a medium such that the instructions are readily executable byspecialized or general purpose hardware, a procedure or method forexecuting the functions, or a combination of any of the above.

The present description and claims may make use of the terms “a,” “atleast one of,” and “one or more of” with regard to particular featuresand elements of the illustrative embodiments. It should be appreciatedthat these terms and phrases are intended to state that there is atleast one of the particular feature or element present in the particularillustrative embodiment, but that more than one can also be present.That is, these terms/phrases are not intended to limit the descriptionor claims to a single feature/element being present or require that aplurality of such features/elements be present. To the contrary, theseterms/phrases only require at least a single feature/element with thepossibility of a plurality of such features/elements being within thescope of the description and claims.

Moreover, it should be appreciated that the use of the term “component,”if used herein with regard to describing embodiments and features of theinvention, is not intended to be limiting of any particularimplementation for accomplishing and/or performing the actions, steps,processes, etc., attributable to and/or performed by the engine. Ancomponent may be, but is not limited to, software, hardware and/orfirmware or any combination thereof that performs the specifiedfunctions including, but not limited to, any use of a general and/orspecialized processor in combination with appropriate software loaded orstored in a machine readable memory and executed by the processor.Further, any name associated with a particular component is, unlessotherwise specified, for purposes of convenience of reference and notintended to be limiting to a specific implementation. Additionally, anyfunctionality attributed to a component may be equally performed bymultiple components, incorporated into and/or combined with thefunctionality of another component of the same or different type, ordistributed across one or more components of various configurations.

In addition, it should be appreciated that the following descriptionuses a plurality of various examples for various elements of theillustrative embodiments to further illustrate example implementationsof the illustrative embodiments and to aid in the understanding of themechanisms of the illustrative embodiments. These examples are intendedto be non-limiting and are not exhaustive of the various possibilitiesfor implementing the mechanisms of the illustrative embodiments. It willbe apparent to those of ordinary skill in the art in view of the presentdescription that there are many other alternative implementations forthese various elements that may be utilized in addition to, or inreplacement of, the examples provided herein without departing from thespirit and scope of the present invention.

The illustrative embodiments may be utilized in many different types ofdata processing environments. In order to provide a context for thedescription of the specific elements and functionality of theillustrative embodiments, FIGS. 1-3 are provided hereafter as exampleenvironments in which aspects of the illustrative embodiments may beimplemented. It should be appreciated that FIGS. 1-3 are only examplesand are not intended to assert or imply any limitation with regard tothe environments in which aspects or embodiments of the presentinvention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

FIGS. 1-3 are directed to describing an example Question Answering (QA)system (also referred to as a Question/Answer system or Question andAnswer system), methodology, and computer program product with which themechanisms of the illustrative embodiments are implemented. The QAsystem is described as an example of a natural language (NL) processingsystem. As will be discussed in greater detail hereafter, theillustrative embodiments are integrated in, augment, and extend thefunctionality of these QA mechanisms with regard to automatic corpusselection and halting condition detection for semantic asset expansion.

Thus, it is important to first have an understanding of how question andanswer creation in a QA system is implemented before describing how themechanisms of the illustrative embodiments are integrated in and augmentsuch QA systems. It should be appreciated that the QA mechanismsdescribed in FIGS. 1-3 are only examples and are not intended to stateor imply any limitation with regard to the type of QA mechanisms withwhich the illustrative embodiments are implemented. Many modificationsto the example QA system shown in FIGS. 1-3 may be implemented invarious embodiments of the present invention without departing from thespirit and scope of the present invention.

As an overview, a Question Answering system (QA system) is an artificialintelligence application executing on data processing hardware thatanswers questions pertaining to a given subject-matter domain presentedin natural language. The QA system receives inputs from various sourcesincluding input over a network, a corpus of electronic documents orother data, data from a content creator, information from one or morecontent users, and other such inputs from other possible sources ofinput. Data storage devices store the corpus of data. A content creatorcreates content in a document for use as part of a corpus of data withthe QA system. The document may include any tile, text, article, orsource of data for use in the QA system. For example, a QA systemaccesses a body of knowledge about the domain, or subject matter area,e.g., financial domain, medical domain, legal domain, etc., where thebody of knowledge (knowledgebase) can be organized in a variety ofconfigurations, e.g., a structured repository of domain-specificinformation, such as ontologies, or unstructured data related to thedomain, or a collection of natural language documents about the domain.

Content users input questions to the QA system which then answers theinput questions using the content in the corpus of data by evaluatingdocuments, sections of documents, portions of data in the corpus, or thelike. When a process evaluates a given section of a document forsemantic content, the process can use a variety of conventions to querysuch document from the QA system, e.g., sending the query to the QAsystem as a well-formed question which are then interpreted by the QAsystem and a response is provided containing one or more answers to thequestion. Semantic content is content based on the relation betweensignifiers, such as words, phrases, signs, and symbols, and what theystand for, their denotation, or connotation. In other words, semanticcontent is content that interprets an expression, such as by usingnatural language processing.

As will be described in greater detail hereafter, the QA system receivesan input question, analyzes the question to extract the major elementsof the question, uses the extracted element to formulate queries, andthen applies those queries to the corpus of data. Based on theapplication of the queries to the corpus of data, the QA systemgenerates a set of hypotheses, or candidate answers to the inputquestion, by looking across the corpus of data for portions of thecorpus of data that have some potential for containing a valuableresponse to the input question. The QA system then performs deepanalysis, e.g., English Slot Grammar (ESG) and Predicate ArgumentStructure (PAS) builder, on the language of the input question and thelanguage used in each of the portions of the corpus of data found duringthe application of the queries using a variety of scoring algorithms.There may be hundreds or even thousands of scoring algorithms applied,each of which performs different analysis, e.g., comparisons, naturallanguage analysis, lexical analysis, or the like, and generates a score.For example, some scoring algorithms may look at the matching of termsand synonyms within the language of the input question and the foundportions of the corpus of data. Other scoring algorithms may look attemporal or spatial features in the language, while others may evaluatethe source of the portion of the corpus of data and evaluate itsveracity.

In the illustrative embodiment, the QA system performs natural languageprocessing based on an ontology. In computer science and informationscience, an ontology is a formal naming and definition of the types,properties, and interrelationships of the entities that really orfundamentally exist for a particular domain of discourse. A lexicon isthe vocabulary of a person, language, or branch of knowledge (such aslegal or medical). In linguistics, a lexicon is an inventory of lexemes.Linguistic theories generally regard human languages as consisting oftwo parts: a lexicon, essentially a catalogue of words, and a grammar, asystem of rules which allow for the combination of those words intomeaningful sentences.

The illustrative embodiments provide mechanisms for selecting corporafor expansion of the lexicon or other semantic assets. The illustrativeembodiments then perform lexicon expansion by finding terms in aselected corpus not previously present in the lexicon and then addingthe new terms to the lexicon. The illustrative embodiments also providemechanisms for detecting a halting condition when new terms are unlikelyto be discovered with another iteration using another corpus. Thus, theillustrative embodiments halts lexicon expansion when the estimatedyield does not warrant the time and processing resources necessary forprocessing another corpus.

The scores obtained from the various scoring algorithms indicate theextent to which the potential response is likely to be a correct answerto the input question based on the specific area of focus of thatscoring algorithm. Each resulting score is then weighted against astatistical model, which is used to compute the confidence that the QAsystem has regarding the evidence for a candidate answer being thecorrect answer to the question. This process is repeated for each of thecandidate answers until the QA system identifies candidate answers thatsurface as being significantly stronger than others and thus, generatesa final answer, or ranked set of answers, for the input question.

As mentioned above, QA systems and mechanisms operate by accessinginformation from a corpus of data or information (also referred to as acorpus of content), analyzing it, and then generating answer resultsbased on the analysis of this data. Accessing information from a corpusof data typically includes: a database query that answers questionsabout what is in a collection of structured records, and a search thatdelivers a collection of document links in response to a query against acollection of unstructured data (text, etc.). Conventional questionanswering systems are capable of generating answers based on the corpusof data and the input question, verifying answers to a collection ofquestions from the corpus of data, and selecting answers to questionsfrom a pool of potential answers, i.e. candidate answers.

Content creators, such as article authors, electronic document creators,web page authors, document database creators, and the like, determineuse cases for products, solutions, and services described in suchcontent before writing their content. Consequently, the content creatorsknow what questions the content is intended to answer in a particulartopic addressed by the content. Categorizing the questions, such as interms of roles, type of information, tasks, or the like, associated withthe question, in each document of a corpus of data allows the QA systemto more quickly and efficiently identify documents containing contentrelated to a specific query. The content may also answer other questionsthat the content creator did not contemplate that may be useful tocontent users. The questions and answers may be verified by the contentcreator to be contained in the content for a given document. Thesecapabilities contribute to improved accuracy, system performance,machine learning, and confidence of the QA system. Content creators,automated tools, or the like, annotate or otherwise generate metadatafor providing information usable by the QA system to identify thesequestion and answer attributes of the content.

Operating on such content, the QA system generates answers for inputquestions using a plurality of intensive analysis mechanisms whichevaluate the content to identify the most probable answers, i.e.candidate answers, for the input question. The most probable answers areoutput as a ranked listing of candidate answers ranked according totheir relative scores or confidence measures calculated duringevaluation of the candidate answers, as a single final answer having ahighest ranking score or confidence measure, or which is a best match tothe input question, or a combination of ranked listing and final answer.

FIG. 1 depicts a schematic diagram of one illustrative embodiment of aquestion/answer creation (QA) system 100 in a computer network 102. Oneexample of a question/answer generation which may be used in conjunctionwith the principles described herein is described in U.S. PatentApplication Publication No. 2011/0125734. The QA system 100 isimplemented on one or more computing devices 104, 105, 107 (comprisingone or more processors and one or more memories, and potentially anyother computing device elements generally known in the art includingbuses, storage devices, communication interfaces, and the like)connected to the computer network 102. The network 102 includes multiplecomputing devices 104, 105, 107 in communication with each other andwith other devices or components via one or more wired and/or wirelessdata communication links, where each communication link comprises one ormore of wires, routers, switches, transmitters, receivers, or the like.The QA system 100 and network 102 enables question/answer (QA)generation functionality for one or more QA system users via theirrespective computing devices 110, 112. Other embodiments of the QAsystem 100 may be used with components, systems, sub-systems, and/ordevices other than those that are depicted herein.

The QA system 100 is configured to implement a QA system pipeline 108that receive inputs from various sources. For example, the QA system 100receives input from the network 102, a corpus of electronic documents106, QA system users, and/or other data and other possible sources ofinput. In one embodiment, some or all of the inputs to the QA system 100are routed through the network 102. The various computing devices 104,105, 107 on the network 102 include access points for content creatorsand QA system users. Some of the computing devices 104, 105, 107 includedevices for a database storing the corpus of data 106 (which is shown asa separate entity in FIG. 1 for illustrative purposes only). Portions ofthe corpus of data 106 may also be provided on one or more other networkattached storage devices, in one or more databases, or other computingdevices not explicitly shown in FIG. 1. The network 102 includes localnetwork connections and remote connections in various embodiments, suchthat the QA system 100 may operate in environments of any size,including local and global, e.g., the Internet.

In one embodiment, the content creator creates content in a document ofthe corpus of data 106 for use as part of a corpus of data with the QAsystem 100. The document includes any file, text, article, or source ofdata for use in the QA system 100. QA system users access the QA system100 via a network connection or an Internet connection to the network102, and input questions to the QA system 100 that are answered by thecontent in the corpus of data 106, In one embodiment, the questions areformed using natural language. The QA system 100 analyzes and interpretsthe question, and provides a response to the QA system user, e.g., QAsystem user 110, containing one or more answers to the question. In someembodiments, the QA system 100 provides a response to users in a rankedlist of candidate answers while in other illustrative embodiments, theQA system 100 provides a single final answer or a combination of a finalanswer and ranked listing of other candidate answers.

The QA system 100 implements a QA system pipeline 108 which comprises aplurality of stages for processing an input question and the corpus ofdata 106. The QA system pipeline 108 generates answers for the inputquestion based on the processing of the input question and the corpus ofdata 106. The QA system pipeline 108 will be described in greater detailhereafter with regard to FIG. 3.

In some illustrative embodiments, the QA system 100 may be the IBMWatson™ system available from International Business MachinesCorporation of Armonk, New York, which is augmented with the mechanismsof the illustrative embodiments described hereafter. As outlinedpreviously, the IBM Watson™ QA system receives an input question whichit then analyzes to extract the major features of the question, that inturn are then used to formulate queries that are applied to the corpusof data. Based on the application of the queries to the corpus of data,a set of hypotheses, or candidate answers to the input question, aregenerated by looking across the corpus of data for portions of thecorpus of data that have some potential for containing a valuableresponse to the input question. The IBM Watson™ QA system then performsdeep analysis on the language of the input question and the languageused in each of the portions of the corpus of data found during theapplication of the queries using a variety of scoring algorithms. Thescores obtained from the various scoring algorithms are then weightedagainst a statistical model that summarizes a level of confidence thatthe IBM Watson™ QA system has regarding the evidence that the potentialresponse, i.e., candidate answer, is inferred by the question. Thisprocess is repeated for each of the candidate answers to generate rankedlisting of candidate answers which may then be presented to the userthat submitted the input question, or from which a final answer isselected and presented to the user. More information about the IBMWatson™ QA system may be obtained, for example, from the IBM Corporationwebsite, IBM Redbooks, and the like. For example, information about theIBM Watson™ QA system can be found in Yuan et al., “Watson andHealthcare,” IBM developerWorks, 2011 and “The Era of Cognitive Systems:An Inside Look at IBM Watson and How it Works” by Rob High, IBMRedbooks, 2012.

FIG. 2 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments are implemented. Data processingsystem 200 is an example of a computer, such as server 104 or client 110in FIG. 1, in which computer usable code or instructions implementingthe processes for illustrative embodiments of the present invention arelocated. In one illustrative embodiment, FIG. 2 represents a servercomputing device, such as a server 104, which, which implements a QAsystem 100 and QA system pipeline 108 augmented to include theadditional mechanisms of the illustrative embodiments describedhereafter.

In the depicted example, data processing system 200 employs a hubarchitecture including north bridge and memory controller hub (NB/MCH)202 and south bridge and input/output (I/O) controller hub (SB/ICH) 204.Processing unit 206, main memory 208, and graphics processor 210 areconnected to NB/MCH 202. Graphics processor 210 is connected to NB/MCH202 through an accelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 212 connectsto SB/ICH 204. Audio adapter 216, keyboard and mouse adapter 220, modem222, read only memory (ROM) 224, hard disk drive (HDD) 226, CD-ROM drive230, universal serial bus (USB) ports and other communication ports 232,and PCI/PCIe devices 234 connect to SB/ICH 204 through bus 238 and bus240. PCI/PCIe devices may include, for example, Ethernet adapters,add-in cards, and PC cards for notebook computers. PCI uses a card buscontroller, while PCIe does not. ROM 224 may be, for example, a flashbasic input/output system (BIOS).

HDD 226 and CD-ROM drive 230 connect to SB/ICH 204 through bus 240. HDD226 and CD-ROM drive 230 may use, for example, an integrated driveelectronics (IDE) or serial advanced technology attachment (SATA)interface. Super I/O (SIO) device 236 is connected to SB/ICH 204.

An operating system runs on processing unit 206. The operating systemcoordinates and provides control of various components within the dataprocessing system 200 in FIG. 2. As a client, the operating system is acommercially available operating system such as Microsoft® Windows 8®.An object-oriented programming system, such as the Java™ programmingsystem, may run in conjunction with the operating system and providescalls to the operating system from Java™ programs or applicationsexecuting on data processing system 200.

As a server, data processing system 200 may be, for example, an IBM®eServer™ System p® computer system, running the Advanced InteractiveExecutive (AIX®) operating system or the LINUX® operating system. Dataprocessing system 200 may be a symmetric multiprocessor (SMP) systemincluding a plurality of processors in processing unit 206.Alternatively, a single processor system may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 226, and are loaded into main memory 208 for execution byprocessing unit 206. The processes for illustrative embodiments of thepresent invention are performed by processing unit 206 using computerusable program code, which is located in a memory such as, for example,main memory 208, ROM 224, or in one or more peripheral devices 226 and230, for example.

A bus system, such as bus 238 or bus 240 as shown in FIG. 2, iscomprised of one or more buses. Of course, the bus system may beimplemented using any type of communication fabric or architecture thatprovides for a transfer of data between different components or devicesattached to the fabric or architecture. A communication unit, such asmodem 222 or network adapter 212 of FIG. 2, includes one or more devicesused to transmit and receive data. A memory may be, for example, mainmemory 208, ROM 224, or a cache such as found in NB/MCH 202 in FIG. 2.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIGS. 1 and 2 may vary depending on the implementation.Other internal hardware or peripheral devices, such as flash memory,equivalent non-volatile memory, or optical disk drives and the like, maybe used in addition to or in place of the hardware depicted in FIGS. 1and 2. Also, the processes of the illustrative embodiments may beapplied to a multiprocessor data processing system, other than the SMPsystem mentioned previously, without departing from the spirit and scopeof the present invention.

Moreover, the data processing system 200 may take the form of any of anumber of different data processing systems including client computingdevices, server computing devices, a tablet computer, laptop computer,telephone or other communication device, a personal digital assistant(PDA), or the like. In some illustrative examples, data processingsystem 200 may be a portable computing device that is configured withflash memory to provide non-volatile memory for storing operating systemfiles and/or user-generated data, for example. Essentially, dataprocessing system 200 may be any known or later developed dataprocessing system without architectural limitation.

FIG. 3 illustrates a QA system pipeline for processing an input questionin accordance with one illustrative embodiment. The QA system pipelineof FIG. 3 may be implemented, for example, as QA system pipeline 108 ofQA system 100 in FIG. 1. It should be appreciated that the stages of theQA system pipeline shown in FIG. 3 are implemented as one or moresoftware engines, components, or the like, which are configured withlogic for implementing the functionality attributed to the particularstage. Each stage is implemented using one or more of such softwareengines, components or the like. The software engines, components, etc.are executed on one or more processors of one or more data processingsystems or devices and utilize or operate on data stored in one or moredata storage devices, memories, or the like, on one or more of the dataprocessing systems. The QA system pipeline of FIG. 3 is augmented, forexample, in one or more of the stages to implement the improvedmechanism of the illustrative embodiments described hereafter,additional stages may be provided to implement the improved mechanism,or separate logic from the pipeline 300 may be provided for interfacingwith the pipeline 300 and implementing the improved functionality andoperations of the illustrative embodiments.

As shown in FIG. 3, the QA system pipeline 300 comprises a plurality ofstages 310-380 through which the QA system operates to analyze an inputquestion and generate a final response. In an initial question inputstage, the QA system receives an input question 310 that is presented ina natural language format. That is, a user inputs, via a user interface,an input question 310 for which the user wishes to obtain an answer,e.g., “Who were Washington's closest advisors?” In response to receivingthe input question 310, the next stage of the QA system pipeline 300,i.e. the question and topic analysis stage 320, analyzes the inputquestion using natural language processing (NLP) techniques to extractmajor elements from the input question, and classify the major elementsaccording to types, e.g., names, dates, or any of a plethora of otherdefined topics. For example, in the example question above, the term“who” may be associated with a topic for “persons” indicating that theidentity of a person is being sought, “Washington” may be identified asa proper name of a person with which the question is associated,“closest” may be identified as a word indicative of proximity orrelationship, and “advisors” may be indicative of a noun or otherlanguage topic.

In addition, the extracted major features include key words and phrasesclassified into question characteristics, such as the focus of thequestion, the lexical answer type (LAT) of the question, and the like.As referred to herein, a lexical answer type (LAT) is a word in, or aword inferred from, the input question that indicates the type of theanswer, independent of assigning semantics to that word. For example, inthe question “What maneuver was invented in the 1500s to speed up thegame and involves two pieces of the same color?,” the LAT is the string“maneuver.” The focus of a question is the part of the question that, ifreplaced by the answer, makes the question a standalone statement. Forexample, in the question “What drug has been shown to relieve thesymptoms of ADD with relatively few side effects?,” the focus is “drug”since if this word were replaced with the answer, e.g., the answer“Adderall” can be used to replace the term “drug” to generate thesentence “Adderall has been shown to relieve the symptoms of ADD withrelatively few side effects.” The focus often, but not always, containsthe LAT. On the other hand, in many cases it is not possible to infer ameaningful LAT from the focus.

Referring again to FIG. 3, the identified major elements of the questionare then used during a hypothesis generation stage 340 to decompose thequestion into one or more search queries that are applied to the corporaof data/information 345 in order to generate one or more hypotheses.Hypothesis generation stage includes primary search phase 341 andcandidate answer generation stage 342. In primary search phase 341, thequeries are generated in any known or later developed informationretrieval (IR) query language, such as the Indri, Lucene, or the like.The queries are applied to one or more text indexes storing informationabout the electronic texts, documents, articles, websites, and the like,that make up the corpora of data/information 345. That is, these varioussources themselves, different collections of sources, and the like,represent a different corpus within the corpora 345. There may bedifferent corpora defined for different collections of documents basedon various criteria depending upon the particular implementation. Forexample, different corpora may be established for different topics,subject matter categories, sources of information, or the like. As oneexample, a first corpus may be associated with healthcare documentswhile a second corpus may be associated with financial documents.Alternatively, one corpus may be documents published by the U.S.Department of Energy while another corpus may be IBM Redbooks documents.Any collection of content having some similar attribute may beconsidered to be a corpus within the corpora 345.

The queries are applied to one or more text indexes storing informationabout the electronic texts, documents, articles, websites, and the like,that make up the corpus of data/information, e.g., the corpus of data106 in FIG. 1. The queries are applied to the corpus of data/informationat the hypothesis generation stage 340 to generate results identifyingpotential hypotheses for answering the input question, which can then beevaluated. That is, the application of the queries results in theextraction of portions of the corpus of data/information matching thecriteria of the particular query. These portions of the corpus are thenanalyzed and used, during the hypothesis generation stage 340, togenerate hypotheses for answering the input question 310. Thesehypotheses are also referred to herein as “candidate answers” for theinput question. For any input question, at this stage 340, there may behundreds of hypotheses or candidate answers generated that may need tobe evaluated.

The QA system pipeline 300, in stage 350, then performs a deep analysisand comparison of the language of the input question and the language ofeach hypothesis or “candidate answer,” as well as performs evidencescoring to evaluate the likelihood that the particular hypothesis is acorrect answer for the input question. There are various ways ofgenerating such scores depending upon the particular analysis beingperformed. In generally, however, these algorithms look for particularterms, phrases, or patterns of text that are indicative of terms,phrases, or patterns of interest and determine a degree of matching withhigher degrees of matching being given relatively higher scores thanlower degrees of matching.

Thus, for example, an algorithm may be configured to look for the exactterm from an input question or synonyms to that term in the inputquestion, e.g., the exact term or synonyms for the term “movie,” andgenerate a score based on a frequency of use of these exact terms orsynonyms. In such a case, exact matches will be given the highestscores, while synonyms may be given lower scores based on a relativeranking of the synonyms as may be specified by a subject matter expert(person with knowledge of the particular domain and terminology used) orautomatically determined from frequency of use of the synonym in thecorpus corresponding to the domain. Thus, for example, an exact match ofthe term “movie” in content of the corpus (also referred to as evidence,or evidence passages) is given a highest score. A synonym of movie, suchas “motion picture” may be given a lower score but still higher than asynonym of the type “film” or “moving picture show.” Instances of theexact matches and synonyms for each evidence passage may be compiled andused in a quantitative function to generate a score for the degree ofmatching of the evidence passage to the input question.

Thus, for example, a hypothesis or candidate answer to the inputquestion of “What was the first movie?” is “The Horse in Motion.” If theevidence passage contains the statements “The first motion picture evermade was ‘The Horse in Motion’ in 1878 by Eadweard Muybridge. It was amovie of a horse running,” and the algorithm is looking for exactmatches or synonyms to the focus of the input question, i.e. “movie,”then an exact match of “movie” is found in the second sentence of theevidence passage and a highly scored synonym to “movie,” i.e. “motionpicture,” is found in the first sentence of the evidence passage. Thismay be combined with further analysis of the evidence passage toidentify that the text of the candidate answer is present in theevidence passage as well, i.e. “The Horse in Motion.” These factors maybe combined to give this evidence passage a relatively high score assupporting evidence for the candidate answer “The Horse in Motion” beinga correct answer.

It should be appreciated that this is just one simple example of howscoring can be performed. Many other algorithms of various complexitiesmay be used to generate scores for candidate answers and evidencewithout departing from the spirit and scope of the present invention.

In accordance with the illustrative embodiments, the above topicanalysis, hypothesis generation, and hypothesis and evidence scoring instages 320-350 are performed using semantic assets 341. In oneembodiment, semantic assets 341 comprise a lexicon, taxonomy, ordictionary with known terms or words and known relationships betweenterms. Organizations spend considerable time, effort, and resources tocraft quality lexicons intended for their analytics processes. Thequality of results from the natural language processing, semanticanalysis, and analytics processes used in stages 320-350 depend to alarge extent on the coverage of the lexicons in corpora 345. In largecorpora, achieving good coverage is considerably difficult and timeconsuming. A typical investment may involve a dozen people working forseveral days just to refine a small set of lexicons.

The ability to accurately gauge lexicon coverage and identify optimalcorpora 345 used to form the semantic assets 341 then becomesincreasingly important, because such metrics allow the organization toassess the time and effort for producing semantic assets 341 by avoidingextra unnecessary effort while maximizing the impact or coverage of thelexicons. More importantly, this acts as a proxy to estimate the qualityand accuracy of downstream natural language processing, semanticanalysis, and analytics processes. Mechanisms for automatic corpusselection and halting condition detection for expansion of semantic aredescribed below with reference to FIGS. 4-9.

In answer ranking stage 360, the scores generated by the various scoringalgorithms are synthesized into confidence scores or confidence measuresfor the various hypotheses. This process involves applying weights tothe various scores, where the weights have been determined throughtraining of the statistical model employed by the QA system and/ordynamically updated. For example, the weights for scores generated byalgorithms that identify exactly matching terms and synonyms may be setrelatively higher than other algorithms that evaluate publication datesfor evidence passages.

The weighted scores are processed in accordance with a statistical modelgenerated through training of the QA system that identifies a manner bywhich these scores may be combined to generate a confidence score ormeasure for the individual hypotheses or candidate answers. Thisconfidence score or measure summarizes the level of confidence that theQA system has about the evidence that the candidate answer is inferredby the input question, i.e. that the candidate answer is the correctanswer for the input question.

The resulting confidence scores or measures are processed by answerranking stage 360, which compares the confidence scores and measures toeach other, compares them against predetermined thresholds, or performsany other analysis on the confidence scores to determine whichhypotheses/candidate answers are the most likely to be the correctanswer to the input question. The hypotheses/candidate answers areranked according to these comparisons to generate a ranked listing ofhypotheses/candidate answers (hereafter simply referred to as “candidateanswers”).

Supporting evidence collection phase 370 collects evidence from evidenceretrieval 371 that supports the candidate answers from answer rankingphase 360. From the ranked listing of candidate answers in stage 360 andsupporting evidence from supporting evidence collection stage 370, QAsystem pipeline 300 generates a final answer, confidence score, andevidence 380, or final set of candidate answers with confidence scoresand supporting evidence, and outputs answer, confidence, and evidence380 to the submitter of the original input question 310 via a graphicaluser interface or other mechanism for outputting information.

FIG. 4 is a block diagram illustrating a system for semantic assetexpansion in accordance with an illustrative embodiment. Semantic assetexpansion system 410 comprises corpus selection component 411, termpicking component 412, and halt condition detection component 413.Semantic asset expansion system 410 generates and expands semanticassets 420 from one or more of corpora 401-404.

In an illustrative embodiment, corpus selection component 411 selects acorpus among corpora 401-404 to generate semantic assets 420, In a firstpass, any corpus will be likely to generate a number of new terms and/orrelationships for semantic assets 420, such as a lexicon, dictionary, ortaxonomy. Term picking component 412 adds words or terms to semanticassets 420. For example, term picking component 412 may use approachessuch as Word2vec or GLIMPSE to create semantic assets. Halt conditiondetection component 413 determines whether a number of new terms from anext corpus warrants sufficient for another iteration of semantic assetexpansion.

in one embodiment, semantic asset expansion system 410 starts with alexicon L made up of terms, T₁ . . . T_(N). Semantic asset expansionsystem 410 then queries a given corpus in corpora 401-404 to get a termfrequency count for each known term T₁ . . . T_(N) in semantic assets420 to obtain frequency counts Semantic asset expansion system 410 sortsF₁ . . . F_(N) in descending order.

FIG. 5 shows an example graph of frequency counts for known terms in agiven corpus in accordance with an illustrative embodiment. The graphshown in FIG. 5 is a very simple example for illustrative purposes. Toaccount for outliers, semantic asset analysis system 410 selects amiddle portion using a heuristic. An example heuristic is as follows:

Let start=0.4×N

Let stop=0.6×N

If (start−stop)==0 then stop=start+1

The above heuristic is an example, and the values and other aspects maybe modified depending on the implementation. Also, other heuristics maybe used to select a representative portion of the term graph to accountfor outliers. In the depicted example shown in FIG. 5, N=11, start=4,and stop=7. Semantic asset expansion system 410 fits a line to theentries in F_(start) . . . F_(stop) in the graph. In one embodiment,semantic asset expansion system 410 uses least squares fitting to fitthe line to the entries in F_(start) . . . F_(stop) (in log space).Semantic asset expansion system 410 determines the X-axis intercept.This is the lower bound estimate for total words N′ in the lexicon onthe given corpus. Semantic asset expansion system 410 then determines anestimate of the number of new words or terms, ΔN, in the given corpus asfollows ΔN=N′−N.

FIG. 6 shows an example graph of frequency counts for known terms ingiven corpus with a line fitted to find the X-axis intercept inaccordance with an illustrative embodiment. Because the terms with thehighest frequency counts are the most common terms, it is unlikely thatanother corpus will result in new terms with a higher frequency countthan the selected middle portion. In the example depicted in FIG. 6, theline fitted to the selected portion has an X-axis intercept, that isthree terms beyond the known terms (ΔN=3). Note that FIG. 6 represents asimple example. An actual example would include many more terms and mayhave a more complex (e.g., higher order) line fit to the selected middleportion.

In one embodiment, corpus selection component 411 determines theestimated number of new terms in each corpus 401-404. Corpus selectioncomponent 411 sorts corpora 401-404 by the number of new terms andselects the corpus with the highest estimated number of new terms as thenext corpus for term picking component 412 to use for semantic assetexpansion.

Halt condition detection component 413 determines whether the estimatednumber of new terms for the next corpus in the sorted corpora warrantsanother iteration of semantic asset expansion. In one embodiment, haltcondition detection component 413 compares the estimated number of newterms for the next corpus to a threshold and halts semantic assetexpansion if the estimated number of new terms is below that threshold.In an example embodiment, the threshold may be determined based on acost/benefit analysis comparing a resource and time cost of runninganother iteration to a number of estimated number of new terms thatwould result. As an example, one may set the threshold to be one (1)such that the semantic asset expansion system performs another iterationof semantic asset expansion as long as at least one new term is likelyto result. However, due to the resource cost of running the semanticexpansion algorithms, a higher value for the threshold may be selectedto increase the benefit relative to the cost.

In one embodiment, corpus selection 411 may repeat the above calculationfor the estimated number of new terms for all corpora 401-404 and sortcorpora 401-404 for each iteration. Each corpus 401-404 may have adifferent set of new terms and the new terms added to semantic assets420 in the last iteration may alter the sort order of corpora 401-404.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

FIG. 7 is a flowchart illustrating operation of a system for automaticcorpus selection and halting condition detection for semantic assetexpansion in accordance with an illustrative embodiment. Operationbegins (block 700), and the system sorts corpora in an order to beconsidered for semantic asset expansion (block 701). In one embodiment,the system may sort corpora in descending order of estimated number ofnew terms in each corpus, as illustrated below with reference to FIG. 8.

The system considers the first (next) corpus (block 702). The systemdetermines an estimated number of missing terms in the corpus (block703). Operation of determining the estimated number of missing terms ina corpus is described below with reference to FIG. 9. The system thendetermines whether the estimated number of missing terms is greater thana threshold (block 704). If the estimated number of missing terms isgreater than the threshold, then the system performs semantic assetexpansion using the corpus (block 705).

The system then determines whether the corpus is the last corpus in thecorpora (block 706). If the currently considered corpus is not the lastcorpus, then operation returns to block 702 to consider the next corpus.If the currently considered corpus is the last corpus in block 706 orthe estimated number of new terms in the current corpus is not greaterthan the threshold in block 704, then the system halts semantic assetexpansion and operation ends (block 707).

FIG. 8 is a flowchart illustrating operation of a mechanism for sortingcorpora by estimated number of new terms in accordance with anillustrative embodiment. Operation begins (block 800), and the mechanismconsiders the first (next) corpus (block 801). The mechanism determinesan estimated number of missing terms in the corpus (block 802).Operation of determining the estimated number of missing terms in acorpus is described below with reference to FIG. 9. The mechanismdetermines whether the corpus is the last corpus in the corpora (block803). If the currently considered corpus is not the last corpus, thenoperation returns to block 801 to consider the next corpus. If thecorpus is the last corpus in block 803, then the mechanism sorts thecorpora by estimated number of missing terms (block 804). Thereafter,operation ends (block 805).

FIG. 9 is a flowchart illustrating operation of a mechanism fordetermining an estimated number of missing terms in a corpus inaccordance with an illustrative embodiment. Operation begins (block900), and the mechanism obtains term frequency counts in the corpus forknown terms (block 901). The mechanism sorts the known terms byfrequency in descending order (block 902). The mechanism then selects amiddle portion of the known terms to eliminate outliers (block 903). Anexample heuristic for selecting a middle portion is as follows:

Let start=0.4×N

Let stop=0.6×N

If (start−stop)==0 then stop=start+1

The above heuristic is an example, and the values and other aspects maybe modified depending on the implementation. Also, other heuristics maybe used to select a representative portion of the term graph to accountfor outliers.

The mechanism then fits a line to the selected entries in the middleportion of the known terms (block 904). The mechanism determines anX-axis intercept of the line (block 905). The mechanism determines theestimated number of missing terms based on the X-axis intercept (block906). Thereafter, operation ends (block 907).

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

A first multi-instance learned model is trained based on whether apassage set returned for a given candidate answer collectively does ordoes not support a correct answer. A second multi-instance learned modelis for instance-level classification and tries to distill what, in theindividual passage feature vectors, led to the overall judgement of thepolarity of the passage set label. Between the two models, the QA systempipeline has the complementary capabilities for answer ranking andpassage ranking. A supporting evidence collection stage scores theevidence and offers the user justification as appropriate for thecandidate answer chosen.

As noted above, it should be appreciated that the illustrativeembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one example embodiment, the mechanisms of theillustrative embodiments are implemented in software or program code,which includes but is not limited to firmware, resident software,microcode, etc.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers, Network adapters mayalso be coupled to the system to enable the data processing system tobecome coupled to other data processing systems or remote printers orstorage devices through intervening private or public networks. Modems,cable modems and Ethernet cards are just a few of the currentlyavailable types of network adapters.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The embodiment was chosen and described in order to bestexplain the principles of the invention, the practical application, andto enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated. The terminology used hereinwas chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

1. A method, in a data processing system comprising at least oneprocessor and at least one memory, the at least one memory comprisinginstructions executed by the at least one processor to cause the atleast one processor to implement an automated lexicon expansion for anidentified corpus, the method comprising: for a selected corpus in a setof corpora, determining an estimated number of new terms in the selectedcorpus that are not in the lexicon based on a frequency count of knownterms in the selected corpus; responsive to the estimated number of newterms in the selected corpus being greater than a threshold, performinglexicon expansion using the selected corpus to form an expanded lexicon;and responsive to the estimated number of new terms in the selectedcorpus not being greater than the threshold, halting lexicon expansion.2. The method of claim 1, wherein determining the estimated number ofnew terms in the selected corpus comprises: identifying a set of knownterms in the lexicon; for each known term in the lexicon, identifying anassociated frequency of occurrence of the known term in the selectedcorpus; sorting the set of known terms based on the associated frequencyof occurrence thereby forming a sorted set of known terms; fitting aline to a portion of the sorted set of known terms; determining anX-axis intercept of the line; and determining the estimated number ofnew terms in the selected corpus based on the X-axis intercept of theline.
 3. The method of claim 2, wherein determining the estimated numberof new terms in the given corpus further comprises subtracting a numberof terms in the sorted set of known terms from the X-axis intercept. 4.The method of claim 1, further comprising: for each given corpus in theset of corpora, determining an estimated number of new terms in thegiven corpus that are not in the lexicon based on a frequency countknown terms in the given corpus; and sorting the set of corpora byestimated number of new terms to form a sorted set of corpora.
 5. Themethod of claim 4, further comprising selecting a corpus having ahighest estimated number of new terms as the selected corpus for lexiconexpansion.
 6. The method of claim 4, wherein determining the estimatednumber of new terms in the given corpus comprises: identifying a set ofknown terms in the lexicon; for each known term in the lexicon,identifying an associated frequency of occurrence of the known term inthe given corpus; sorting the set of known terms based on the associatedfrequency of occurrence thereby forming a sorted set of known terms;fitting a line to a portion of the sorted set of known terms;determining an X-axis intercept of the line; and determining theestimated number of new terms in the given corpus based on the X-axisintercept of the line. 7-18. (canceled)